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SUMMARY 


By concentrating on defining and improving specific Configuration Management (CM) 
functions, processes, procedures, personnel selection/development, and tools, 
internal and external customers received improved CM services. Job performance 
within the section increased in both satisfaction and output. Participation in 
achieving major improvements has led to the delivery of consistent quality CM 
products as well as significant decreases in every measured CM metrics category. 


GETTING STARTED 

In early 1989, the Network Control center Data System (NCCDS) Configuration 
Management Section was composed of two full-time technical people, one technical 
person on loan (to be used as required), one task leader, and the section 
manager. People had been in these positions for two-three years and knew their 
jobs. The section manager was new to the company, but not to the CM function, 
the software/engineering field, nor to Total Quality Management (TQM) . The main 
functions of the CM group are to: 

- Provide support to formal project reviews, and baseline and control 
documentation 

- Support configuration item identification and discrepancy reporting 
system activities 

- Maintain software product baselines 

- Control changes to various software releases at different testing 
levels 

- Provide status, accounting, reporting, and traceability 

- Conduct internal audits and support formal project audits 

- Coordinate, track, and report Data Management function activities 

The challenge was to "coach” the CM group into one which recognized all of the 
above responsibilities and responded with quality output to the NCCDS community, 
consistently. 
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WHAT WAS IT LIKE BEFORE IMPROVEMENT? 


In order to fully appreciate the tremendous gains that have taken place in the 
CM section, a little time must be devoted to understanding where the section 
needed improvement. The major areas were: 

o Section Characteristics 

o Procedures 

o Tools 

o Communication 

Section Characteristics: There were 3.0 staff to support over 100 people project 

wide, which produced a total of 450k DSI for the NCCDS system. Although the 
staff was working in the CM functional area, most were only familiar with the 
product control aspect. There was no CM status, reports, involvement with the 
Technical Review Board (TRB) or Configuration Review Board (CRB), no 
documentation reviews, and no emphasis on quality of work at every level. The 
task leader was the only person with a college degree and the only person who 
knew most all machine platforms as well as being able to troubleshoot and analyze 
CM problems. The task leader was the only person who was cross trained and could 
step in and help out all areas in addition to helping out during crisis 
situations. The hours for all personnel were long and frustrating, with little 
praise for good work. CM had the responsibility to support 7 different software 
segments (CCS, gnss, its, nfe, NTS, RAP, and SPS), on 4 different hardware 
platforms (VAX, UNISYS 1100, MASSCOMF, and Intel architecture), in 2 facility 
areas: The Development Test & Training ( DT&T ) and Operations, The Section 

Manager, although experienced and knowledgeable of the CM function, was new to 
the company and new to the nccds. Emphasis on training CM personnel or improving 
CM processes did not seem to be a priority. 

Procedures: Of the 7 segments which CM supported , only 4 systems had any 

written procedures. Three of these procedure sets were poorly written, 
incomplete, and incorrect in several areas. The other set of procedures were 
more of a history of the segment, rather than procedures needed to perform 
routine functions of that segment. There were few clear steps to follow in any 
sequential order. Because a new software segment was being developed, there were 
no procedures in that segment. With no staff assigned to that CM segment on a 
full time basis, there was little emphasis to write CM procedures for that 
segment. There were many ideas, troubleshooting mechanisms, tips, procedures, 
and methods written on sheets of paper gathered in notebooks which tended to be 
lost easily. The procedures that were documented were inconsistently written 
across segments. This did not support staff cross training. There was also no 
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one place which housed all CM procedures. worse, few people used the correct 
procedures which did exist. 

Tools: Simply stated, most tools which were available to the CM group at that 
time did not work. Custom made tool sets were not maintained thereby causing 
errors when unknowing staff used them. There was no time scheduled to 
investigate the root causes and correct problems, just time to fix them. There 
were many laborious work-arounds that staff used because automated routines were 
not available or the ability to keep them current did not exist. The 
inefficiencies resulted in long processing times, incorrect output, and longer 
fix times. The mere difficulty in using some of the tools themselves caused 
errors. These internal CM problems were having enormous effects on the rest of 
the project, in terms of schedule, reliability, cost, causing staff frustration 
and lowering confidence in the ability of CM to do the job. 

Communication: During this time, CM processing time requirements were not 
recognized on any official project schedules. The time CM required was discussed 
in management meetings, although internal schedules never reflected the resource. 
The section manager discussed with the development and test managers the need to 
"steal” a day on each end of "their" schedules to accommodate CM requirements. 
This method of acquiring schedule was not conducive to smooth transitions. Most 
times, the software deliveries were made on the last day of their schedule at 
6:00pm and test expected to start the next day at 6:00am. There was no routine 
status accounting or reporting to the project of CM units processed, reports 
tracking documentation, or CM efficiency and productivity. In addition, there 
was little input from CM to the overall project planning process, needs, and 
problem areas. Participation in CSC Project Management System (PMS) planning, 
weekly reporting of CM activities to the Assistant Technical Representatives 
(ATRs), and monthly presentations to GSFC management of CM 
accomplishments /problems areas was weak. 

WHAT WAS DONE? 


There were two efforts undertaken to improve the CM function: 1) A management 
initiative to improve section processes and routine ways of conducting business, 
and 2) Establishment of process improvements through the participation of CM 
team members in the Task Oriented process Improvement Committee (TOPIC). 

MANAGEMENT INITIATIVES: After assessing the situation, management devoted 
emphasis to 1) staffing 2) project participation 3) defining procedures 4) 
improving tools 5) providing status and reports, and 6) self evaluation of CM 
processes, several areas were totally re-engineered. 
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To remedy the staffing situation, over the next year and a half, there were 8-9 
staff personnel hired to work in the CM function. An additional person was on 
loan, part time, to assist with tools and CM sponsored a summer hire, who helped 
with the CM data base development. At the end of the CM personnel transition, 
all personnel had completed their bachelors, 3 people had completed their 
masters, 3 people were working on their masters, and 1 person was working on a 
PHd. This higher level of educated personnel was then applied to every segment, 
which allowed a different degree of work to be performed. In reality, this 
transition of personnel took nearly 2 years to evolve, and has never stopped. 
The increased capabilities of the personnel have allowed a much easier cross 
training of different personnel on different software segments. It reinforced 
the necessity to have educated and trained personnel and precipitated regular 
training for CM including internal classes, SEAS courses, vendor classes (both 
brought into CSC and attendance on vendor sites), and attendance at conferences. 
The higher level of personnel expertise enabled CM to be able to analyze, 
troubleshoot, and resolve problems within our own section. People who were doing 
the work and making errors were able to begin fixing them. This also enabled the 
group to have insight into what and where some of the root causes of the problems 
were. 

As CM began working more with project management, quality assurance, release 
leaders, and other technical people, the need for CM to identify processing time 
on schedules became a reality. Internal schedules contained references to CM 
time required as well as providing detailed planning schedules prepared by 
release leaders used to plan Integration, System Test, Acceptance Test, and 
Operations transitions. CM personnel were able to plan for work and knew what 
the project deadlines were and where CM fit into the big picture. CM also began 
to schedule machine/facility resources in the Software Development Facility 
( SDF ) , the DT&T, Emergency NCC (ENCC), and Operations areas. By this time the 
NCCDS had added two facilities; one a development facility in the CSC Greentec 
I area, and the other an ENCC at the GSFC facility. This added to the CM 
responsibility of maintaining equal configurations for each release. Having to 
maintain multiple releases at different test levels sometimes necessitated that 
CM have their own machine time to perform some processing and installation 
functions. CM therefore began to schedule resources in the required facilities 
and "piggy-backed" off other's scheduled time when there was no CM or other 
function impact. 

Procedures were another area which improved dramatically. Personnel have 
documented or updated all CM procedures. Procedure formats were standardized 
across all segments in a logical step by step fashion. They were also written 
to be user friendly, incorporating helpful processing notes. Today the 
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procedures are used as a training tool for new CM personnel • They are also 
updated on a routine basis, as a part of the CSC Program Management System (PMS) 
accountability system. In addition to the documented detailed procedures, an 
NCCDS CM Software Plan was developed and is revised periodically. 

Two major efforts were undertaken to improve CM tools used on the CCS and SPS 
segments • 


1. In 1989 a study was undertaken to determine the best CM tool to use on 

the VAX. After the investigation was complete a presentation and report 
was provided to GSFC. The decision was to continue to use the existing 
CSC written tool which would be enhanced and coupled with Digital 
Equipment Corporations MMS (Module Management System) tool. Task 

personnel basically completed the VAX tool effort in 1990. Upgrades have 
been added each year to continue improving tool productivity and 
efficiency. 

2. Another effort was undertaken in 1990 to improve the Unisys tool. 
Although there were off-the-shelf 
tools evaluated, none provided the 
control, reporting, and speed that 
were desired. Over a period of one 
year, task personnel first 
identified the areas which required 
immediate attention and made the 
proper fixes. Second, desired 
enhancements were identified and 
gradually added. Both the fixes 
and enhancements were applied in an 
internal controlled manner. 

Internal Problem Reports were 
written and resolution 
recommendations were evaluated. Various fixes and enhancements were 
packaged together and released in builds to the tool. The build was first 
tested in a testbed on the SDF Unisys. After all bugs had been 
eliminated, it was inserted into the regular controlled tool which was 
used in the SDF. Figure 1 contains a high level overview of the SPS tool 
fixes and enhancements. A summary was presented to the project CRB, 
approved and then applied to the DT&T controlled version of the tool. 
These enhancements allowed the tool to run more efficiently and later 
several utilities were automated into a menu program. 


SPS TOOL 

FIXES: 

0 Modified code to work on Unisys 2200 
0 Modified code to correctly assign level dependent files 
0 Modified code to generate cross-reference tables 
0 Modified code to correctly identified source code type 

ENHANCEMENTS: 

0 Modified code to allocate mass storage efficiently 
0 Developed code to add diagnostic error messages 
0 Developed routines to summarize proccesslng results 
0 Developed routines to check entire baseline 


Figure I 
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The benefits as a result of the tool improvements have been noticeable throughout 
the section and the project. No longer are incorrect software versions of a unit 
delivered to test. Listings are routinely run, reviewed, and archived for 
traceability purposes. These are used later for troubleshooting, if required. 
The two-three day test sessions have not been hindered by the inability of CM to 
locate an incorrectly assigned level dependent file, should there be a problem, 
listings with diagnostic error messages assist in locating the source of the 
problem quickly and easily. 


93.1 SPRs at LVL 2 System Test 

WEEKLY FROM 12/04/92 - 4/30/93 


Another tool 
enhancement to the 
section was the 



creation of a CM 
relational data base 
to provide 
traceability, reports , 
and on line 
information. Up to 
this time there was no 
routine accountability 
of CM activities to 
the project. There 
were no reports or 
statistics kept within 
the section. Task 


Figure 2 


personnel developed a 
data base to house CM 


data from the internal Software Delivery Forms (ISDF) and data regarding the CM 
processing. Data such as segment, subsystem, type of delivery, ISPR/SPR/STR 
number, unit name, type of unit, date received, and date processed was collected 
and entered. A units processed report for each release is provided to the 
project each week. Another report showing elapsed time indicates CM efficiency 
in processing deliveries from the time of receipt to the time available at any 
level for testing. Graphs are produced at each project phase for each release 
and show what (if any) CM problems are occurring on each segment. Figure 2 shows 
the CM problems by segment for Release 9 3.1 during the System Test phase. These 
weekly graphs act as a catalyst for internal CM Defect Causal Analysis (DCA) and 
in continuing process improvement. This database eventually was merged with the 
System Engineering Project Database ( SEDB ) and is known today as the 
Configuration Management Database ( CMDB ) and is maintained and has been improved 
by the System Engineering database section. 


CM has code counter tools for each of the NCCDS segments. Over the last two 
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years, each tool has been upgraded to be more efficient, easier to use, and has 
been fixed to reduce errors. All of the tool improvements have enabled the CM 
section to provide products to both internal and external customers faster, 
better, cheaper, and more accurately. 

THE CM TASK ORIENTED PROCESS IMPROVEMENT COMMITTEE (TOPIC): At the same time the 

above management initiatives were taking place, the CM group began setting aside 
a small amount of time each week to discuss changes in the section that would 
improve quality, productivity, worker satisfaction, and reduce errors. In 1990, 
with cscs increased interest in Total Quality Management, the group became known 
as the Task Oriented Process improvement Committee (TOPIC). The section manager 
sponsored the group, and every six-eight months the group chose a new 
facilitator. 

The first project the group undertook 
was to design a set of checklists 
which were to be used as a 
verification tool and used in 
conjunction with the processing 
procedures. Although the processing 
procedures were being revised, the 
group wanted a high level composite 
list of "musts" that should be 
accomplished that a second person 
could verify to ensure the proper 
steps had been followed. Each segment 
lead prepared a set of "checks" for 
each segment and each processing 
phase. For example, the CCS checklist 
for processing a delivery includes 
seven "checks". The verifier 

completes the checklist as the actual 
item is checked. The verification is 
a combination of checking on the 
terminals and checking the printouts. 

The checklists can be used for test 
levels 1, 2, and 3 and are signed and dated by the verifier. An example of an 
SPS checklist is shown in Figure 3. Other checklists have been designed for 
phases such as: 1) Creating a Baseline, 2) Processing a Delivery, 3) SDF/DDT 

Transfer Tape Update, 4) Creating Failover Tapes, 5) Installing a Release, 6) 
Making Operational Tapes, 7) Processing Symbolics, Procs, Schema, Templates, Maps 
and QLP Reports, 8) Updating a Baseline, 9) Installing a Release Into OPS, and 
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10) Deleting Units. The checklists are tailored to incorporate segment 
differences. 

Another process established to be used with the checklists, was the incorporation 

of a reviewer. Usually, the reviewer 
is either the task leader or the 
section manager. The review session 
takes place prior to the delivery or 
product being installed in the SDF. 
The processor, verifier, and reviewer 
go over the delivery from beginning to 
end to ensure all steps have been 
completed properly and without error. 
All printouts, listings, checklists, 
and original delivery paperwork are 
reviewed and retained by the CM lead 
for the segment. Any future inquiries 
into a delivery, can be recovered and 
investigated if required. This 

three-pronged approach has added discipline to the overall process and assisted 
greatly in the reduction of errors. A partial list of CM TOPIC achievements is 
shown in Figure 4. 

The TOPIC also initiated what they called a "shake down" test. CM processes all 
the deliveries for a given release for a particular group (i.e Integration Test, 
System Test, or Acceptance Test) to begin testing. After the installation is 
made on either the SDF or the DT&T machines, but prior to turnover to the test 
group, CM coordinates a "shake down" test. This test is a multi discipline team 
effort comprising the CM segment leads, an integration tester or system tester, 
a computer operator, maintenance and data base personnel. The CCS, ITS, and SPS 
segments are brought up, connected, and are checked to ensure all segments "talk" 
to each other. Although no functions are performed, data passed, or test cases 
run, this simple check has pinpointed several errors. These problems were 
cleared up prior to the baseline being provided to the "internal test customer". 
Time is saved by CM and the testing groups and customer satisfaction is enhanced. 


CM TOPIC ACHIEVEMENTS 

• Standard Verification Check Lists 

• Standard Code Count Form 

• Computer Time Usage Log Form 

• Operational Software Installation Form (OSIF) 

• Monthly Backup Forms 

• Delivery Form Error Reductions 

Figure 4 


WHAT WAS IT LIKE AFTER IMPROVEMENTS? 

Specific: There have been many improvements over the past three years. in 
addition to the overall management initiatives and TOPIC achievements there have 
been other individual task improvements. Listed below are only those specific 
improvements which have been documented through either the CSC Code 550 or 530 
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cost avoidance system: 


IMPROVEMENT 

1. Revised NTS Build Procedure 

2. Improved CM Procedures 

3. Simplified Delivery Process 

4. Modified CCS Compilation 
Process 

5. Designed standard DSI Form 

6. Eliminated Duplicate DSI 
Counts and Reduced Errors 

7. Revised and Designed New CM 
Valtab Procedures and Form 


STAFF HOURS -$ SAVINGS /YEAR 

- 24 staff hours 

- 5500 sheets of treated 
paper per year 

- 60 staff hours 

- 32 staff hours 

- 150 staff hours 

- 18 VAX CPU hours in 1991 
98 VAX CPU hours in 1992 
106 VAX CPU hours in 1993 

- 114 staff hours 

- $ 14 , 022 over 3 years 

- $ 5,530 over 3 years 

- 26 staff hours 


Statistical: Because either little or partial data was maintained in the late 
80' s in the CM section, the best possible attempt has been made to present fair 
and accurate data. Emphasis has been placed over the last few years on quality 
deliveries to our internal customers • Some of those internal customers are 
Integration and System Test. 

During the integration Testing phase, integration Software Problem Reports 
(ISPRs) are written to document problems. Available data show there was a 50% 
reduction in errors over the three builds after the 89.1 Release series. From 
Release 90.1 to Release 93.1, CM ISPR errors decreased to around 11%. During the 
Release 3 Build 0 integration testing, there were zero CM errors out of a total 
of 51 problems. 

CM was accountable for nearly 20% of the project's Software Problem Reports 
(SPRs), written during System Test phase, up through the 89.1 Release series. 
Figure 5 shows the SPR trend for earlier releases. During the time many of the 
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improvements were initiated. Release 
90.1 was being developed and CM SPRs 
fell to about 13%. This trend 
continued into the development of 
Release 91. lA, and CM SPRs dropped to 
under 6%. By the time Release 92.1 
was being System Tested, CM SPRs were 
down to just over 4%. 

More recently, as shown in Figure 6, 
the trend has continued to be the 
same. Release 93.0 found CM SPRs at 
zero. Release 93.1 was a much larger 



development effort and CM SPRs were 
around 2.5%. Release 3 Build 0 is 
currently under test and to date, cm 
SPR s are .8%. Clearly, the number of 
errors attributable to CM has 
decreased substantially. obviously, 
the time spent in correcting problems 
has decreased accordingly, and a much 
greater confidence level has been 
achieved from groups receiving CM 
products. The project can now count 
on CM to make internal schedules. 

Figure 6 


Figure 7 shows a composite of the 
number of total problems against the 
number of CM problems . 

This improvement has also been seen in 
installations and products delivered 
to GSFC • During the Release 89.1 
series of deliveries to Acceptance 
Test, CM accounted for over 13% of the 
errors identified in the release. 
Acceptance Test documents problems on 
a System Trouble Report (STR). Over 
the next three releases, problems 


CURRENT PROBLEM TREND 



RELEASE 


CM PROBLEMS — - TOTAL PROBLEMS 
Figure 7 
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attributable to CM fell to less than 2.5%, There were also no CM STRs for 
Release 9 3.0 and only one CM STR for Release 9 3.1.1, which went operational 
recently. The turn around in the percentage of CM problems has been dramatic 
over the past few years. The quality and dependability of CM products and 
services to our customers has been increased by these measurable results. 

Participation: Key to the improvements has been the acceptance of all CM team 

members to want to make a difference and to make things better. Early in the 
process, team members recognized the need to become more efficient and more 
productive. As the opportunity became available to participate in TQM committees 
all CM section members took advantage. The contributions to the CM TOPIC through 
the years has been directly responsible for many of the CM successes and 
improvements. CM has had 100% participation in the five major TQM committees. 
Two of the five first committees were facilitated by CM members. All CM team 
members have been involved in Process Improvement Committee (PIC) Process Action 
Teams (PATs). This participation across project functions to improve a process 
has provided team members with insight into resolving multi disciplined problems 
which benefit everyone. The enthusiasm and willingness of CM team members to 
participate at all levels of TQM activities has strengthened the project, the 
section, and the individuals involved. Everyone wins. 

Recognition: When a job needs to be done, it should not be done to seek 

recognition. Over time, as each year rendered better results, individuals within 
the CM group and the team as a whole realized technical and professional 
recognition. Listed below are some of those achievements: 

o Documented and received four Flight Dynamics Quality Improvement Ledgers 
citing success stories 

o Many of the CM achievements have been publicized in the SEAS Total 
Quality Management Highlights 

o Individual CM members and the CM team have received NCC Awards and 
Recognition Committee (ARC) monthly recognition certificates 

o Documented and received three recent cost avoidance success reports 

o Two CM team members received FDTG Engineering Employee of the 
Year Awards 

o One CM team member was honored as the first recipient of the NCC Project 
Dedication, Adaptability, Team Spirit, Unique Solutions and Motivation 
(DATUM) Award 

o One CM team member was a winner of the SEAS TQM Involvement Award 

The team was also nominated for the 1992 SEAS Quality Service Award and an 
individual nominated for the 1993 NTG Quality Service Award. In addition, there 
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have been letters of commendation from other CSC codes and from the GSFC customer 
on CM team members excellent service and support. 

CONCLUSION 


This paper has listed many CM improvements over a wide spectrum and shown 
meaningful statistical evidence of positive results. The above findings, 
however, do not mean the group is perfect or that the job is done. The challenge 
is to provide "continuous” improvements. Because the gap has been tremendously 
narrowed, future improvements will probably not be measured in whole percentages. 
The hard job will be to continue to chip away until the goal is obtained. The 
goal is to have zero processing errors, to provide internal and external 
customers CM products and services which are error free, and to continue to 
increase CM efficiency and productivity. "In CM, we don't make the software... 
we make it better I” 
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